DISRPT 2023
Shared Task
News 04/05/23: data have been updated (fixed mistakes in .rels of surprise datasets TEDm and CRPC), don't forget to pull the new data files.
News: deadline extension, systems and papers are due on May, 14
News: Test and surprise data are now available in our repository: https://github.com/disrpt/sharedtask2023
News 17/04/23: data have been updated, don't forget to pull the new data files.
Please check our FAQ page for more information about participation, evaluation etc.!
Study of coherence relations in frameworks such as RST (Mann & Thompson 1988), SDRT (Asher & Lascarides 2003) and PDTB (Miltsakaki et al. 2004), has experienced a revival in the last few years, in English and many other languages (Matthiessen & Teruya 2015; Maziero et al. 2015; da Cunha 2016; Iruskieta et al. 2016; Zeldes 2016, 2017). Multiple sites are now actively engaged in the development of discourse parsers (Lin et al. 2014, Feng and Hirst 2014; Ji and Eisenstein 2014; Joty et al. 2015; Surdeanu et al. 2015; Xue et al. 2016; Braud et al. 2017; Guz & Carenini 2020; kobayashi et al. 2020; Nguyen et al. 2021; Kobayashi et al. 2021; Zhao et al. 2021; Yu et al. 2022; Atwell et al. 2022; Kurfalı 2022; Nishida and Matsumoto 2022; Huber et al. 2022), as a goal in itself, but also for applications such as sentiment analysis, argumentation mining, summarization, question answering, or machine translation evaluation (Benamara et al., 2017; Gerani et al. 2019; Durrett et al. 2016; Peldszus & Stede 2016; Scarton et al. 2016; Schouten & Frasincar 2016; Xu et al. 2020; among many others). At the same time, evaluation of results in discourse parsing has proven complicated (see Morey et al. 2017), and progress in integrating results across discourse treebanking frameworks has been slow. We propose shared tasks on discourse unit segmentation, connective identification and discourse relation classification across formalisms (see below) that aim to promote convergence of resources and a joint evaluation of discourse parsing approaches, following the two first campaigns (Zeldes et al. 2019; Zeldes et al. 2021).
Description of the Tasks
Tasks 1 and 2 were initially proposed in 2019. You can consult the results here and the associated papers here. Task 3 was first introduced in 2021. You can view the last results for all tasks here and the associated papers here.
TASK 1: Discourse Unit Segmentation across Formalisms
The DISRPT 2019 workshop introduced the first iteration of a cross-formalism shared task on discourse unit segmentation. Since all major discourse parsing frameworks imply a segmentation of texts into segments, learning segmentations for and from diverse resources is a promising area for converging methods and insights. We provide training, development, and test datasets from all available languages and treebanks in the RST, SDRT and PDTB formalisms, using a uniform format. Because different corpora, languages and frameworks use different guidelines for segmentation, the shared task is meant to promote design of flexible methods for dealing with various guidelines, and help to push forward the discussion of standards for discourse units. For datasets which have treebanks, we will evaluate in two different scenarios: with and without gold syntax, or otherwise using provided automatic parses for comparison.
TASK 2: Discourse Connective Identification across Languages
The DISRPT 2019 workshop also provided the first iteration of a shared task on cross-lingual connective identification, using data annotated within the PDTB framework. We repeat the task in 2021 with updated datasets aiming to identify the location of discourse connectives indicating the presence of coherence relations.
TASK 3: Discourse Relation Classification across Formalisms
We also continue the first iteration of a cross-formalism shared task on discourse relation classification in 2021. Data is converted from three distinct, but overlapping frameworks: Rhetorical Structure Theory (RST, Mann & Thompson 1988), Penn Discourse Treebank (PDTB, Miltsatsaki et al. 2004), and Segmented Discourse Representation Theory (SDRT, Asher & Lascarides 2003). The goal of the shared task is to bring together diverse strands of research on discourse relation identification, which are sometimes siloed separately due to differences between underlying data structures and assumptions of different frameworks. In order to enable approaches benefiting from multiple datasets created using distinct points of view, the task aims to find a common denominator in representing all available datasets, for the widest possible range of languages.
Shared Task Data and Formats
Data for the shared task will be soon released via GitHub together with format documentation and tools. A sample of the data is already available on: https://github.com/disrpt/sharedtask2023.
Schedule:
25 January 2023 – Sample data released
22 February 2023 – Train / dev data release
17 April 2023 – Test data release
14 May 2023 8 May 2023 – Submission of system and paper
26 May 2023 22 May 2023 - Notification of acceptance
5 June 2023 1 June 2023 - Camera-ready paper due
13-14 July 2023 - CODI Workshop at ACL
Google group for participants, please join us on: disrpt2023_participants@googlegroups.com to receive e-mail updates whenever new data is made available for the shared task.
Discord group for participants, please join us on: https://discord.gg/JDdjhXaK
To contact the organisers, you can also send an email to disrpt_chairs@googlegroups.com.
References
Asher, Nicholas, and Alex Lascarides. 2003. Logics of Conversation. Cambridge: Cambridge University Press.
Atwell, K., Sicilia, A., Hwang, S. J., & Alikhani, M. (2022, May). The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error. In Findings of the Association for Computational Linguistics: ACL 2022 (pp. 824-845).
Benamara, Farah, Maite Taboada & Yannick Mathieu. 2017. Evaluative language beyond bags of words: Linguistic insights and computational applications. Computational Linguistics 43(1), 201–264.
Braud, Chloé, Maximin Coavoux & Anders Søgaard. 2017. Cross-lingual RST discourse parsing. Proceedings of EACL 2017. Valencia, Spain, 292–304.
da Cunha, Iria. 2016. Towards discourse parsing in Spanish. Papers presented at TextLink - Structuring Discourse in Multilingual Europe - Second Action Conference. Budapest, Hungary.
Durrett, Greg, Taylor Berg-Kirkpatrick & Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, 1998–2008.
Feng, Vanessa Wei & Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. Proceedings of ACL 2014. Baltimore, MD, 511–521.
Gerani, Shima, Giuseppe Carenini & Raymond Ng. 2019. Modeling content and structure for abstractive review summarization. Computer Speech and Language.
Guz, Grigorii and Giuseppe Carenini. 2020. Corefer-ence for Discourse Parsing: A Neural Approach.In Proceedings of the First Workshop on Computa-tional Approaches to Discourse, pages 160–167, On-line. Association for Computational Linguistics.
Huber, P., Xing, L., & Carenini, G. 2022. Predicting above-sentence discourse structure using distant supervision from topic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 10, pp. 10794-10802).
Iruskieta, Mikel, Gorka Labaka & Juliano Desiderato Antonio. 2016. Detecting the central units in two different genres and languages: A preliminary study of Brazilian Portuguese and Basque texts. Procesamiento del Lenguaje Natural 56, 65–72.
Ji, Yangfeng and Jacob Eisenstein. 2014. Representation Learning for Text-Level Discourse Parsing. In Proceedings of ACL 2014, pages 13–24, Baltimore, MD.
Joty, Shafiq, Giuseppe Carenini & Raymond Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics 41(3), 385–435.
Kobayashi, N., Hirao, T., Kamigaito, H., Okumura, M., & Nagata, M. (2020, April). Top-down RST parsing utilizing granularity levels in documents. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 05, pp. 8099-8106).
KOBAYASHI, Naoki, HIRAO, Tsutomu, KAMIGAITO, Hidetaka, et al. Improving neural RST parsing model with silver agreement subtrees. In : Proceedings of NAACL HLT 2021. p. 1600-1612.
Kurfalı, M. (2022). Contributions to Shallow Discourse Parsing: To English and beyond (Doctoral dissertation, Department of Linguistics, Stockholm University).
Lin, Z., Ng, H., & Kan, M. (2014). A PDTB-Styled End-to-End Discourse Parser. Nat. Lang. Eng., 20, 151-184.
Mann, William C., and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse 8(3), 243–281.
Matthiessen, Christian M.I.M. & Kazuhiro Teruya. 2015. Grammatical realizations of rhetorical relations in different registers. Word 61(3), 232–281.
Maziero, Erick G., Graeme Hirst & Thiago A. S. Pardo. 2015. Semi-supervised never-ending learning in rhetorical relation identification. Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria.
Miltsakaki, Eleni, Rashmi Prasad, Aravind K. Joshi & Bonnie L. Webber. 2004. The Penn Discourse Treebank. In Proceedings of LREC 2004. Lisbon, Portugal.
Morey, Mathieu, Philippe Muller & Nicholas Asher. 2017. How Much Progress have we Made on RST Discourse Parsing? A Replication Study of Recent Results on the RST-DT. In: Proceedings of EMNLP 2017. Copenhagen, Denmark, 1319–1324.
Nguyen, T. T., Nguyen, X. P., Joty, S., & Li, X. (2021, June). RST Parsing from Scratch. In Proceedings of NAACL HLT, 2021, 1613-1625.
Nishida, N., & Matsumoto, Y. (2022). Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation. Transactions of the Association for Computational Linguistics, 10, 127-144.
Peldszus, Andreas & Manfred Stede. 2016. Rhetorical structure and argumentation structure in monologue text. Proceedings of the 3rd Workshop on Argument Mining, ACL. Berlin, Germany, 103–112.
Riccardi, Giuseppe, Frederic Bechet, Morena Danieli, Benoit Favre, Robert Gaizauskas, Udo Kruschwitz & Massimo Poesio. 2015. The SENSEI Project: Making sense of human conversations. In J. F. Quesada, F. J. Martín Mateos & T. López-Soto (eds.), Future and Emergent Trends in Language Technology. Proceedings of the First International FETLT Workshop. Berlin: Springer, 10–33.
Schouten, Kim & Flavius Frasincar. 2016. COMMIT at SemEval-2016 Task 5: Sentiment analysis with Rhetorical Structure Theory. Proceedings of SemEval-2016. San Diego, CA, 356–360.
Scarton, Carolina, Daniel Beck, Kashif Shah, Karin Sim Smith & Lucia Specia. 2016. Word embeddings and discourse information for Machine Translation Quality Estimation. Proceedings of the First Conference on Machine Translation, ACL. Berlin, Germany, 831–837.
Surdeanu, Mihai, Thomas Hicks & Marco Valenzuela-Escárcega. 2015. Two practical Rhetorical Structure Theory parsers. Proceedings of NAACL 2015. Denver, CO, 1–5.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-Aware Neural Extractive Text Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021–5031, Online. Association for Computational Linguistics.
Xue, Nianwen, Hwee Tou Ng, Sameer Pradhan, Attapol T. Rutherford, Bonnie Webber, Chuan Wang & Hongmin Wang. 2016. CoNLL 2016 Shared Task on multilingual shallow discourse parsing. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, 1–19.
Yu, N., Zhang, M., Fu, G., & Zhang, M. (2022, May). RST Discourse Parsing with Second-Stage EDU-Level Pre-training. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4269-4280).
Zeldes, Amir. 2016. rstWeb: A browser-based annotation interface for Rhetorical Structure Theory and discourse relations. Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2016) System Demonstrations. San Diego, CA, 1–5.
Zeldes, Amir. 2017. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation 51(3), 581–612.
Amir Zeldes, Debopam Das, Erick Maziero Galani, Juliano Desiderato Antonio, and Mikel Iruskieta. 2019. Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019. Association for Computational Linguistics, Minneapolis, MN, edition.
Amir Zeldes, Yang Janet Liu, Mikel Iruskieta, Philippe Muller, Chloé Braud, and Sonia Badene. 2021. Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021). Association for Computational Linguistics, Punta Cana, Dominican Republic, edition.
Zhao, Z., & Webber, B. (2021, November). Revisiting Shallow Discourse Parsing in the PDTB-3: Handling Intra-sentential Implicits. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse (pp. 107-121).